130 research outputs found
Semantics, Modelling, and the Problem of Representation of Meaning -- a Brief Survey of Recent Literature
Over the past 50 years many have debated what representation should be used
to capture the meaning of natural language utterances. Recently new needs of
such representations have been raised in research. Here I survey some of the
interesting representations suggested to answer for these new needs.Comment: 15 pages, no figure
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
Recurrent neural networks (RNNs) stand at the forefront of many recent
developments in deep learning. Yet a major difficulty with these models is
their tendency to overfit, with dropout shown to fail when applied to recurrent
layers. Recent results at the intersection of Bayesian modelling and deep
learning offer a Bayesian interpretation of common deep learning techniques
such as dropout. This grounding of dropout in approximate Bayesian inference
suggests an extension of the theoretical results, offering insights into the
use of dropout with RNN models. We apply this new variational inference based
dropout technique in LSTM and GRU models, assessing it on language modelling
and sentiment analysis tasks. The new approach outperforms existing techniques,
and to the best of our knowledge improves on the single model state-of-the-art
in language modelling with the Penn Treebank (73.4 test perplexity). This
extends our arsenal of variational tools in deep learning.Comment: Added clarifications; Published in NIPS 201
Real Time Image Saliency for Black Box Classifiers
In this work we develop a fast saliency detection method that can be applied
to any differentiable image classifier. We train a masking model to manipulate
the scores of the classifier by masking salient parts of the input image. Our
model generalises well to unseen images and requires a single forward pass to
perform saliency detection, therefore suitable for use in real-time systems. We
test our approach on CIFAR-10 and ImageNet datasets and show that the produced
saliency maps are easily interpretable, sharp, and free of artifacts. We
suggest a new metric for saliency and test our method on the ImageNet object
localisation task. We achieve results outperforming other weakly supervised
methods
Dropout Inference in Bayesian Neural Networks with Alpha-divergences
To obtain uncertainty estimates with real-world Bayesian deep learning
models, practical inference approximations are needed. Dropout variational
inference (VI) for example has been used for machine vision and medical
applications, but VI can severely underestimates model uncertainty.
Alpha-divergences are alternative divergences to VI's KL objective, which are
able to avoid VI's uncertainty underestimation. But these are hard to use in
practice: existing techniques can only use Gaussian approximating
distributions, and require existing models to be changed radically, thus are of
limited use for practitioners. We propose a re-parametrisation of the
alpha-divergence objectives, deriving a simple inference technique which,
together with dropout, can be easily implemented with existing models by simply
changing the loss of the model. We demonstrate improved uncertainty estimates
and accuracy compared to VI in dropout networks. We study our model's epistemic
uncertainty far away from the data using adversarial images, showing that these
can be distinguished from non-adversarial images by examining our model's
uncertainty
- …